Neural Information Processing Systems http://nips.cc/

In this paper the authors propose a novel recurrent convolutional encoder-decoder network for learning to apply out-of-plane rotations to 3d objects such as human faces and 3d chair models. The proposed network starts from a basic model, where its encoder network disentangles the input image into identity units and pose units, then with the action units applied on pose units to control the rotation direction, its decoder network which consists of convolution and unsampling decode the identity and pose into an image of rotated object and the corresponding object mask. To support longer rotation trajectories, the proposed network is then extended to have the recurrent architecture where the encoded identity unit of input image is fixed and the pose unit is changed by a sequence of action units, and finally both identity and pose units are fed into decoder to generate the result image. One of main contribution of this paper is learning to disentangle the representations for identity/appearance and pose factors, where the identity units are shown to be a discriminative view-invariant features in the cross-view object recognition task. In addition, this disentangling properties will benefit more and predict better rendering while using the longer rotation trajectories in the curriculum training stages for training the proposed recurrent convolutional encoder-decoder network.

An important problem for both graphics and vision is to synthesize novel views of a 3D object from a single image. This is in particular challenging due to the partial observability inherent in projecting a 3D object onto the image space, and the ill-posedness of inferring object shape and pose. However, we can train a neural network to address the problem if we restrict our attention to specific object classes (in our case faces and chairs) for which we can gather ample training data. In this paper, we propose a novel recurrent convolutional encoder-decoder network that is trained end-to-end on the task of rendering rotated objects starting from a single image. The recurrent structure allows our model to capture longterm dependencies along a sequence of transformations, and we demonstrate the quality of its predictions for human faces on the Multi-PIE dataset and for a dataset of 3D chair models, and also show its ability of disentangling latent data factors without using object class labels.

An important problem for both graphics and vision is to synthesize novel views of a 3D object from a single image. This is in particular challenging due to the partial observability inherent in projecting a 3D object onto the image space, and the ill-posedness of inferring object shape and pose. However, we can train a neural network to address the problem if we restrict our attention to specific object classes (in our case faces and chairs) for which we can gather ample training data. In this paper, we propose a novel recurrent convolutional encoder-decoder network that is trained end-to-end on the task of rendering rotated objects starting from a single image. The recurrent structure allows our model to capture long- term dependencies along a sequence of transformations, and we demonstrate the quality of its predictions for human faces on the Multi-PIE dataset and for a dataset of 3D chair models, and also show its ability of disentangling latent data factors without using object class labels.